Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up build script stuff and documentation. #7215

Merged
merged 1 commit into from
Aug 6, 2019

Conversation

ehuss
Copy link
Contributor

@ehuss ehuss commented Aug 5, 2019

I often get lost trying to understand some of this build script stuff, so this is an attempt to make it a little easier to follow. Overview:

  • Remove Context::build_script_overridden.
    • Add BuildContext::script_override to query if something is overridden.
    • Do not bother to compute unit dependencies for overridden build scripts. This is the most risky change, since there are some subtle assumptions when connecting up the various build script maps. However, as best as I can reason through, it seems correct. The intent here is to avoid all the situations where it attempts to filter out these extra deps.
  • Rename Context::build_state to Context::build_script_outputs to try to be clearer about what it is. Similarly rename BuildState to BuildScriptOutputs.
  • Remove trivial 1-line function load_build_deps.
  • Add some documentation.

@rust-highfive
Copy link

r? @Eh2406

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 5, 2019
@alexcrichton
Copy link
Member

@bors: r+

Great cleanup, thanks!

@bors
Copy link
Contributor

bors commented Aug 6, 2019

📌 Commit bd31c08 has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 6, 2019
@bors
Copy link
Contributor

bors commented Aug 6, 2019

⌛ Testing commit bd31c08 with merge c604416...

bors added a commit that referenced this pull request Aug 6, 2019
Clean up build script stuff and documentation.

I often get lost trying to understand some of this build script stuff, so this is an attempt to make it a little easier to follow. Overview:

- Remove `Context::build_script_overridden`.
    - Add `BuildContext::script_override` to query if something is overridden.
    - Do not bother to compute unit dependencies for overridden build scripts. This is the most risky change, since there are some subtle assumptions when connecting up the various build script maps. However, as best as I can reason through, it seems correct. The intent here is to avoid all the situations where it attempts to filter out these extra deps.
- Rename `Context::build_state` to `Context::build_script_outputs` to try to be clearer about what it is. Similarly rename `BuildState` to `BuildScriptOutputs`.
- Remove trivial 1-line function `load_build_deps`.
- Add some documentation.
@bors
Copy link
Contributor

bors commented Aug 6, 2019

☀️ Test successful - checks-azure
Approved by: alexcrichton
Pushing c604416 to master...

@bors bors merged commit bd31c08 into rust-lang:master Aug 6, 2019
bors added a commit to rust-lang/rust that referenced this pull request Aug 8, 2019
Update cargo

9 commits in 26092da337b948719549cd5ed3d1051fd847afd7..42a8c0adf91323c01228268c651aef5366b25b69
2019-07-31 23:24:32 +0000 to 2019-08-07 20:41:07 +0000
- Add support for customizing JSON diagnostics from Cargo (rust-lang/cargo#7214)
- Bump rustfix (rust-lang/cargo#7221)
- Fix remap-path-prefix from failing. (rust-lang/cargo#7219)
- Clean up build script stuff and documentation. (rust-lang/cargo#7215)
- Remove debug panic in package-features. (rust-lang/cargo#7213)
- Fix an old test. (rust-lang/cargo#7208)
- Remove unused AstBuilder (rust-lang/cargo#7210)
- Revert "Release a jobserver token while locking a file" (rust-lang/cargo#7201)
- improve error message for unmatched prerelease dependencies (rust-lang/cargo#7191)
bors added a commit that referenced this pull request Sep 3, 2019
Basic standard library support.

This is not intended to be useful to anyone. If people want to try it, that's great, but do not rely on this. This is only for experimenting and setting up for future work.

This adds a flag `-Zbuild-std` to build the standard library with a project. The flag can also take optional comma-separated crate names, like `-Zbuild-std=core`. Default is `std,core,panic_unwind,compiler_builtins`.

Closes rust-lang/wg-cargo-std-aware#10.

Note: I can probably break some of the refactoring into smaller PRs if necessary.

## Overview
The general concept here is to use two resolvers, and to combine everything in the Unit graph. There are a number of changes to support this:

- A synthetic workspace for the standard library is created to set up the patches and members correctly.
- Decouple `unit_dependencies` from `Context` to make it easier to manage.
- Add `features` to `Unit` to keep it unique and to remove the need to query a resolver.
- Add a `UnitDep` struct which encodes the edges between `Unit`s. This removes the need to query a resolver for `extern_crate_name` and `public`.
- Remove `Resolver` from `BuildContext` to avoid any confusion and to keep the complexity focused in `unit_dependencies`.
- Remove `Links` from `Context` since it used the resolver. Adjusted so that instead of checking links at runtime, they are all checked at once in the beginning. Note that it does not check links for the standard lib, but it should be safe? I think `compiler-rt` is the only `links`?

I currently went with a strategy of linking the standard library dependencies using `--extern` (instead of `--sysroot` or `-L`). This has some benefits but some significant drawbacks. See below for some questions.

## For future PRs
- Add Cargo.toml support. See rust-lang/wg-cargo-std-aware#5
- Source is not downloaded. It assumes you have run `rustup component add rust-src`. See rust-lang/wg-cargo-std-aware#11
- `cargo metadata` does not include any information about std. I don't know how this should work.
- `cargo clean` is not std-aware.
- `cargo fetch` does not fetch std dependencies.
- `cargo vendor` does not vendor std dependencies.
- `cargo pkgid` is not std-aware.
- `--target` is required on the command-line. This should default to host-as-target.
- `-p` is not std aware.
- A synthetic `Cargo.toml` workspace is created which has to know about things like `rustc-std-workspace-core`. Perhaps rust-lang/rust should publish the source with this `Cargo.toml` already created?
- `compiler_builtins` uses default features (pure Rust implementation, etc.). See rust-lang/wg-cargo-std-aware#15
    - `compiler_builtins` may need to be built without debug assertions, see [this](https://github.com/rust-lang/rust/blob/8e917f48382c6afaf50568263b89d35fba5d98e4/src/bootstrap/bin/rustc.rs#L210-L214). Could maybe use profile overrides.
- Panic issues:
    - `panic_abort` is not yet supported, though it should probably be easy. It could maybe look at the profile to determine which panic implementation to use? This requires more hard-coding in Cargo to know about rustc implementation details.
    - [This](https://github.com/rust-lang/rust/blob/8e917f48382c6afaf50568263b89d35fba5d98e4/src/bootstrap/bin/rustc.rs#L186-L201) should probably be handled where `panic` is set for `panic_abort` and `compiler_builtins`. I would like to get a test case for it. This can maybe be done with profile overrides?
- Using two resolvers is quite messy and causes a lot of complications. It would be ideal if it could only use one, though that may not be possible for the foreseeable future. See rust-lang/wg-cargo-std-aware#12
- Features are hard-coded. See rust-lang/wg-cargo-std-aware#13
- Lots of various platform-specific support is not included (musl, wasi, windows-gnu, etc.).
- Default `backtrace` is used with C compiler. See rust-lang/wg-cargo-std-aware#16
- Sanitizers are not built. See rust-lang/wg-cargo-std-aware#17
- proc_macro has some hacky code to synthesize its dependencies. See rust-lang/wg-cargo-std-aware#18. This may not be necessary if this uses `--sysroot` instead.
- Profile overrides cause weird linker errors.
  That is:
  ```toml
  [profile.dev.overrides.std]
  opt-level = 2
  ```
  Using `[profile.dev.overrides."*"]` works. I tried fiddling with it, but couldn't figure it out.
  We may also want to consider altering the syntax for profile overrides. Having to repeat the same profile for `std` and `core` and `alloc` and everything else would not be ideal.
- ~~`Context::unit_deps` does not handle build overrides, see #7215.~~ FIXED

## Questions for this PR
- I went with the strategy of using `--extern` to link the standard lib. This seems to work, and I haven't found any problems, but it seems risky. It also forces Cargo to know about certain implicit dependencies like `compiler_builtins` and `panic_*`. The alternative is to create a sysroot and copy all the crates to that directory and pass `--sysroot`. However, this is complicated by pipelining, which would require special support to copy `.rmeta` files when they are generated. Let me know if you think I should use a different strategy. I'm on the fence here, and I think using `--sysroot` may be safer, but adds more complexity.
    - As an aside, if rustc ever tries to grab a crate from sysroot that was not passed in via `--extern`, then it results in duplicate lang items. For example, saying `extern crate proc_macro;` without specifying `proc_macro` as a dependency. We could prevent rustc from ever trying by passing `--sysroot=/nonexistent` to prevent it from trying. Or add an equivalent flag to rustc.
- How should this be tested? I added a janky integration test, but it has some drawbacks. It requires internet access. It is slow. Since it is slow, it reuses the same target directory for multiple tests which makes it awkward to work with.
    - What interesting things are there to test?
    - We may want to disable the test before merging if it seems too annoying to make it the default. It requires rust-src to be downloaded, and takes several minutes to run, and are somewhat platform-dependent.
- How to test that it is actually linking the correct standard library? I did tests locally with a modified libcore, but I can't think of a good way to do that in the test suite.
- I did not add `__CARGO_DEFAULT_LIB_METADATA` to the hash. I had a hard time coming up with a test case where it would matter.
    - My only thought is that it is a problem because libstd includes a dylib, which prevents the hash from being added to the filename. It does cause recompiles when switching between compilers, for example, when it normally wouldn't.
    - Very dumb question: Why exactly does libstd include a dylib? This can cause issues (see rust-lang/rust#56443).
    - This should probably change, but I want to better understand it first.
- The `bin_nostd` test needs to link libc on linux, and I'm not sure I understand why. I'm concerned there is something wrong there. libstd does not do that AFAIK.
@ehuss ehuss added this to the 1.38.0 milestone Feb 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants